B&PFileNo. 13567-002 

BERESKIN & PARR UNITED STATES 



Title: METHOD AND SYSTEM FOR 

DETECTING AND CORRECTING 
DEFECTS IN A DIGITAL IMAGE 

Inventors: Edythe Patricia LeFeuvre and 
Rodney D. Hale 



- 1 - 

Title : METHOD AND SYSTEM FOR DETECTING AND CORRECTING 
DEFECTS IN A DIGITAL IMAGE 

Field of the Invention 

[0001] The present invention relates generally to digital image 
processing, and more particularly relates to a method and system for 
detecting red-eye defects in a digital image. 

5 Background of the Invention 

[0002] A digital image is made up of the rows and columns of picture 
elements or "pixels". Image size is typically expressed in terms of the number 
of rows and columns of pixels in an image. Pixels typically occupy a regular 
grid structure. A common image size is 640 columns by 480 rows (307,200 
10 pixels total). 

[0003] The color of each pixel in a color image can be described by a 
combination of three primary colors: Red, Green, and Blue. The color depth 
for each pixel specifies the number of different color levels that any pixel in 
the image can have. Typically color depth is expressed in terms of the number 

15 of bits of resolution used to encode color information. A common color 
resolution is 24 bits. At this resolution 8 bits are used to encode Red intensity, 
8 bits for Green intensity and 8 bits for Blue intensity. Therefore, for each 
color component there are 1? or 256 different intensities ranging from 0 to 
255. 0 indicates an absence of a particular color and 255 indicates that that 

20 particular color has a maximum intensity at that particular pixel. 

[0004] Red-eye is a common problem that occurs in photographs of 

people and animals taken in dimly lit places with a flash. Red-eye results 
when a light from the flash enters the eye and bounces off the capillaries in 
the back of the eye. Most flash pictures are taken in relative darkness when 
25 people's pupils are dilated. This allows light to reflect off the capillaries and 
return to the camera. The capillaries, which are filled with blood, produce a 
reflection with a red glow. Typically this happens when the flash is directly 
above the lens and the subject is looking into the camera. If the pupils are 



-2- 



dilated sufficiently, then red-eye will occur even if the subject is not looking 
directly into the camera. 

[0005] Manual correction of red-eye problems in digital images can be 
time-consuming, as the user must identify the specific pixels to be corrected 
5 and then adjust the color data of these pixels until the desired color is 
achieved. Accordingly, automatic methods of correcting red-eye defects have 
been developed. However, these automatic methods suffer from defects 
themselves. 

[0006] That is, scanning an entire digital photograph for red-eye defects 
10 can be time-consuming and may demand considerable processing power. 
Further, problems arise both from false positives and false negatives - false 
positives when the automatic method erroneously identifies objects in the 
digital photograph as red-eye, and false negatives when the automatic 
method fails to locate actual instances of red-eye. 

15 Summary of the invention 

[0007] In accordance with a first aspect of the invention, there is 
provided a method of correcting a red-eye effect in a digital image provided by 
a cluster of pixels. The method comprises (a) conducting at least one tophat 
operation over each pixel in the digital image to provide a tophat image; (b) 

20 conducting an intensity threshold operation on the tophat image to provide a 
segmentation mask for segmenting objects in the digital image; (c) for each 
segmented object in the segmentation rnask, extracting at least one feature 
from at least one of the segmented object and a border rejgion surrounding 
the segmented object and classifying the segmented object based on the at 

25 least one feature; and (d) for each segmented object in the segmentation 
mask classified as red-eye effect in step (c), correcting the red-eye effect by 
re-coloring the segmented object to generate a corrected image. 

[0008] In accordance with a second aspect of the invention, there is 
provided a system for correcting a red-eye effect in a digital image provided 
30 by a cluster of high intensity pixels. The system comprises a memory for 
storing the digital image; and means for performing the steps of (a) 
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conducting at least one tophat operation over each pixel in the digital image to 
provide a tophat image; (b) conducting an intensity threshold operation on the 
tophat image to provide a segmentation mask for segmenting objects in the 
digital image; (c) for each segmented object in the segmentation mask, 
5 extracting at least one feature from at least one of the segmented object and 
a border region surrounding the segmented object and classifying the 
segmented object based on the at least one feature; and (d) for each 
segmented object in the segmentation mask classified as red-eye effect in 
step (c), correcting the red-eye effect by re-coloring the segmented object to 
10 generate a corrected image. 

[0009] In accordance with a third aspect of the invention, there is 
provided a computer program product for use on a computer system to 
correct a red-eye effect in a digital image defined over a cluster of pixels. The 
computer program product comprises a recording medium and means 

15 recorded on the medium for instructing the computer system to perform the 
steps of (a) conducting at least one tophat operation over each pixel in the 
digital image to provide a tophat image; (b) conducting an intensity threshold 
operation on the tophat image to provide a segmentation mask for 
segmenting objects in the digital image; (c) for each segmented object in the 

20 segmentation mask, extracting at least one feature from at least one of the 
segmented object and a border region surrounding the segmented object and 
classifying the segmented object based on the at least one feature; and (d) for 
each segmented object in the segmentation mask classified as red-eye effect 
in step (c), correcting the red-eye effect by re-coloring the segmented object 

25 to generate a corrected image. 

Brief Description of the Drawings 

[0010] A detailed description of preferred aspects is provided herein 
below with reference to the following drawings, in which 

[0011] Figure 1, in a flowchart, illustrates an automatic method for 

30 detecting and removing red-eye defects in accordance with a preferred aspect 
of the invention; 



[0012] Figure 2a illustrates a large red-eye at full resolution; 

[0013] Figure 2b illustrates a large red-eye at half resolution; 

[0014] Figure 2c illustrates a large red-eye at quarter resolution; 

[0015] Figure 2d illustrates a medium red-eye at full resolution; 

5 [0016] Figure 2e illustrates a medium red-eye at half resolution; 

[0017] Figure 2f illustrates a medium red-eye at quarter resolution; 

[0018] Figure 2g illustrates a small red-eye at full resolution; 

[0019] Figure 2h illustrates a small red-eye at half resolution; 

[0020] Figure 2i illustrates a small red-eye at quarter resolution; 

10 [0021] Figure 3a illustrates an original image of a baby having red-eye; 

[0022] Figure 3b illustrates a segmentation mask derived from the 

image of Figure 3a; 

[0023] Figure 3c illustrates a red-eye mask derived from the 
segmentation mask of Figure 3b; 

15 [0024] Figure 4 illustrates an image having a sub-image drawn around 
a red-eye defect; 

[0025] Figure 5, in accordance with a further preferred embodiment of 
the invention, illustrates a method of locating and correcting the red-eye 
defect in the sub-image of Figure 4; 

20 [0026] Figure 6a illustrates an original object in a pixel grid; 

[0027] Figure 6b illustrates the object of Figure 6a after dilation; 

[0028] Figure 6c illustrates the object of Figure 6a after erosion; 

[0029] Figure 6d illustrates a border region of the object of Figure 6a 

derived from the dilated object of Figure 6b and the eroded object of Figure 
25 6c; and. 

[0030] Figure 7, in a block diagram, illustrates a computer system for 
configuring to implement an embodiment of the invention. 
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D tail d Description of Pr f rr d Asp cts 

[0031] Referring to Figure 1, there is illustrated in a flowchart an 
automatic method for detecting and removing red-eye defects in digital 
photographic images. In step 22, the digital photograph image is provided. In 
5 step 24, a full resolution version of this digital image is analyzed to segment 
small compact red objects within the digital image. A color image can be split 
into three greyscale images where each image contains one of the red, green 
or blue component intensities of that image. Segmentation is performed using 
at least one tophat operation performed on an image. The image to be 

10 segmented may be the red component image. Alternatively, a red-intensified 
image may be generated for segmentation, for example, by subtracting the 
green component image from the red component image. Another alternative 
for segmentation is an inverted red-intensified image in which red areas are 
relatively dark, which can be produced, for example, by subtracting the red 

15 component image from the green component image. A bright tophat operation 
(hb) that can be used to segment high intensity regions is defined as follows 
("Digital Image Processing" by Rafael C. Gonzalez and Richard E. Woods, 
Addison Wesley Publishing, 1992 edition): 

hb = f-((fVg)— g) 

20 where: f is the input image, g is a structuring element (a 

structuring element used is a 3 x 3 pixel square), V indicates one or more 
grayscale image erosions, and — indicates one or more grayscale image 
dilations. 

[0032] Similarly, a dark top hat operation (hd) that can be used to 
25 segment low intensity regions is defined as follows: 

hd = ((f— g) Vg)-f 

where: f is the input image, g is a structuring element (a 
structuring element used is a 3 x 3 pixel square), a/ indicates one or more 
grayscale image erosions, and — indicates one or more grayscale image 
30 dilations. 
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[0033] A greyscale erosion is an operation performed on a greyscale 
image. A greyscale erosion replaces each pixel intensity with the minimum 
intensity of the pixel and its surrounding pixels in the original image. 

[0034] For example: 

the original image: 

543552 16 
36598772 
34 149989 
99887235 

becomes: 

333 32 111 
3 1112 111 
3 1112222 
3 1112 2 2 3 



As described above, each of the numbers in this image could vary between 0 
20 and 255 inclusive - the numbers are restricted to those between 1 and 9 
inclusive for ease of representation. 

[0035] A greyscale dilation replaces each pixel intensity with the 
maximum intensity of the pixel and its surrounding pixels in the original image. 
For example, 

25 the eroded image: 

33332 111 
3 1112 111 

3 1112 222 

30 31112223 

becomes: 
333332 11 

35 33333222 

33122233 
33 122233 

[0036] Bright regions in either the bright or the dark tophat image will 
40 correspond to concentrations of red in the original image. As described 



10 



15 



above, the bright tophat operation involves first an erosion and then a dilation. 
Specifically, in an erosion operation, the intensity value for each pixel is 
changed to equal the minimum intensity value in the 8 pixels surrounding that 
pixel and that pixel itself. Thus, by each erosion operation, the borders of a 
5 pixel cluster of bright intensity will creep inwards. If the object provided by the 
pixels of bright intensity is sufficiently small, or sufficient erosion operations 
are performed, then this region of bright intensity will be eliminated. After the 
erosion operation, a dilation operation is performed. This dilation operation is 
the reverse of the erosion operation. That is, for each pixel to which the 

10 dilation operation is applied, the intensity value for that pixel will be replaced 
with the maximum intensity value for each of the 8 pixels surrounding that 
pixel and that pixel itself. Thus, if any of the original region of high intensity to 
which the erosion operation was applied remains, then the dilation operation 
will result in the borders of this high intensity object expanding outwards once 

15 again. However, if this high intensity object has been completely eliminated 
by the erosion operation, then the dilation operation will not bring it back. 

[0037] As shown in the above bright tophat equation, after equal 
numbers of erosion and dilation operations have been applied to the original 
image, the intensity values for this eroded and dilated image are subtracted 
20 from the corresponding intensity values for the original image, producing a 
tophat image in which bright regions correspond to concentrations of red in 
the original image. 

[0038] Similarly, a dark tophat operation could be performed on an 
inverted red-intensified image in which red areas are relatively dark. As shown 
25 in the above dark tophat equation, after equal numbers of dilation and erosion 
operations have been applied to the original image, the intensity values for the 
original image are subtracted from the corresponding intensity values for the 
dilated and eroded image, producing a tophat image in which bright regions 
correspond to concentrations of red in the original image. 

30 [0039] The tophat image is intensity thresholded such that bright 
regions become objects of interest. Of these objects, only objects with 
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compactness less than a compactness threshold are retained as objects of 
interest. Compactness can be defined as follows: 

compactness = object perimeter / (4 x tc x object area) 

In this equation, both the object perimeter and area are 
5 measured in pixels. 

[0040] By this means, elongated patches of redness are removed to 
produce a small segmentation mask. In this small segmentation mask, the 
locations and shapes of the segmented objects are represented by white 
pixels and all other pixels are black. The single segmented object is defined 
10 as a collection of white pixels connected to each other vertically, horizontally 
or diagonally. 

[0041] In step 26, features are extracted from the digital image for each 
object in the small segmentation mask. The features extracted may include 
the original color of each segmented object, its segmented shape, and its 

15 original color texture as well as the original color and texture of the region 
surrounding the object. For example, color features abstracted might include 
the mean red intensity of the object, the maximum red intensity within the 
object, and the mean green intensity of the object. Shape features might 
include perimeter and compactness, A number of texture features are 

20 calculated by measuring the pixel intensities in the object after an edge filter 
has been used on the image. 

[0042] In step 28, each segmented object is classified based on the 
feature set abstracted in step 26. That is, each segmented object is classified 
based on the probability that the object is a red-eye defect. According to an 

25 aspect of the present invention, the probability that an object is a red-eye 
defect can be determined by calculating the degree of similarity to a red-eye 
paradigm cluster. For example, this probability can be determined by 
calculating the object's quadratic classifier distance to a training red-eye 
feature cluster. The equation for the quadratic classifier distance (Q) is as 

30 follows: 
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Q = Features * A * Features' + b * Features + c 
where: 

Features = the feature vector describing the object to be classified 
5 Features' = the transpose of the feature vector 
A = inv(K_non) - inv(K_defect) 

K_non = covariance matrix of the non defect training class 
K_defect = covariance matrix of the defect training class 
b = 2* inv(K_defect) * m_defect - inv(K_non) * m_non 
10 m_non = mean of the non defect class 
m_defect = mean of the defect class 

c = m_non * inv(K_non) * m_non' - m_defect * inv(K__defect) * m_defect' 

[0043] The probability that an object is a red-eye defect is proportional 
to Q is calculated using the above function. If the probability is higher than a 

15 selected threshold, then the object is classified as a red-eye defect, while if 
the probability is lower than the given threshold, the object is classified as not 
a red-eye defect. The threshold is set at a level that produces an acceptable 
compromise between false positives and false negatives. That is, the 
threshold is set at a level that enables an acceptably high proportion of red- 

20 eye defects to be detected, while minimizing the number of objects that are 
erroneously identified as red-eye defects. This classification process produces 
a small red-eye mask. 

[0044] Preferably, multiple resolutions of the digital photograph image 
are used for speed optimization as full resolution is not required for the 

25 detection of medium and large red-eye objects. There is no limit on the 
number of resolutions that can be used. In a preferred aspect of the invention 
illustrated in Figure 1, three resolutions are used: full resolution, half 
resolution and quarter resolution. For example, a 4000 x 4000 pixel image 
will be analyzed at full resolution (4000 x 4000). at half resolution (2000 x 

30 2000) and at quarter resolution (1000 x 1000). These three resolutions are 
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indicated in Figure 1 by the three separate paths of the flowchart that originate 
from step 22. 

[0045] As shown in Figure 1, the initial image provided in step 22 is re- 
sampled in steps 30 and 40 to provide a half resolution image and a quarter 
5 resolution image respectively. Referring to Figure 2, there are illustrated 
various sizes of red-eye at different resolutions. For example, in Figure 2a a 
large red-eye is shown at full resolution, in Figure 2b a large red-eye is shown 
at half resolution, and in Figure 2c a large red-eye is shown at quarter 
resolution. In Figure 2d, a medium red-eye is shown at full resolution, in 
10 Figure 2e a medium red-eye is shown at half resolution, and in Figure 2f a 
medium red-eye is shown at quarter resolution. In Figure 2g, a small red-eye 
is shown at full resolution, in Figure 2h, a small red-eye is shown at half 
resolution, and in Figure 2i, a small red-eye is shown at quarter resolution. 

[0046] A number of advantages flow from conducting the above 
15 analysis on images at different resolutions. Specifically, for red-eye defects 
above a certain size, such defects can be most efficiently found through 
analysis of a quarter resolution image, rather than a half resolution image or a 
full resolution image, as (1) sufficient pixels showing the red-eye defect are 
present, and (2) fewer pixels need be considered in order to find the red-eye 
20 defect. On the other hand, for small red-eyes. information may be lost in the 
lower resolution images that is required to identify the object as a red-eye 
defect at the required level of probability. In such cases, the full resolution 
image may provide the additional information required to classify the object as 
either a red-eye defect or as not being a red-eye defect. 

25 [0047] Analogous to step 24 described above, in steps 32 and 42 the 

half resolution image and quarter resolution image respectively are analyzed. 
In the case of step 32, medium compact red objects in the half resolution 
digital image are segmented to create a medium segmentation mask. In step 
42, large compact red objects in the quarter resolution digital image are 

30 segmented to create the large segmentation mask. 
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[0048] Optionally, at this point, the three segmentation masks created 
in steps 24, 32 and 42 can be combined such that each object segmented in 
one of these segmentation masks appears in the combined segmentation 
mask. Referring to Figure 3, this process is illustrated in an original image, as 
5 well as in a combined segmentation mask and in a final red-eye mask derived 
from the original image. That is, a small segmentation mask, medium 
segmentation mask and large segmentation mask are created in steps 
analogous to steps 24, 32 and 42 respectively described above. Then, unlike 
the method illustrated in Figure 1, these three segmentation masks are 

10 combined to provide the combined segmentation masks shown in Figure 3b. 
As shown, areas of high intensity redness in the original image, such as the 
red-eyes 50 of the baby 48, the nostrils 52 of the baby 48 and the lips 54 of 
the baby 48 appear as areas of white intensity in the surrounding black of the 
combined segmentation mask of Figure 3b, Then, analogous to step 26 of 

15 Figure 1 described above, features are abstracted from the segmented 
objects as well as their surrounding areas, and the objects identified in the 
combined segmentation masks of Figure 3b are classified as red-eye defects 
or as not being red-eye defects in a step analogous to step 28, thereby 
providing the final red-eye mask shown in Figure 3c. In Figure 3c, the lips, 

20 nostrils and other objects shown in the combined segmentation mask have 
been discarded as they do not meet a selected threshold probability of being 
instances of red-eye, such that the only white areas remaining correspond to 
the red-eyes 50a of the baby 48. 

[0049] The flowchart of Figure 1 does not illustrate this method of 
25 Figure 3. Instead, the feature abstraction steps and classification steps are 
separately executed with respect to each segmentation mask. Specifically, in 
step 34, analogous to step 36, features are abstracted from the segmented 
objects in the medium segmentation mask provided in step 32. Similarly, in 
step 44. features are abstracted from the objects segmented and their 
30 surrounding areas in the large segmentation mask created in step 42. Then, 
in steps 36 and 46 respectively, objects segmented in the medium 
segmentation mask and large segmentation mask respectively are classified 
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as red-eye if they meet the threshold probability - that is, if each object's 
quadratic classifier distance is sufficiently small to meet the probability 
threshold selected. 

[0050] In steps 38 and 48, the medium red-eye mask provided in step 
5 36 and the large red-eye mask provided in step 46 respectively are resized to 
full resolution. Then, in step 50, the red-eye masks provided by steps 28. 38 
and 48 are disjunctively combined to yield a final red-eye mask having each 
red-eye object in each of the red-eye masks. In step 52. objects classified as 
red-eye are re-colored to produce a corrected image. That is, an object 

10 classified as red-eye is re-colored to remove its red appearance. In a first 
stage, the object classified as red-eye is re-colored in the original image so 
that each pixel within the object is provided with new color values based on 
the color of the corresponding pixel in the original image. For example, the 
new red color could be set to equal the average of the original green color 

15 value and the original blue color value. The new green color value could be 
set to equal the new red color value minus 10 (to a minimum of 0). The new 
blue color value could be set to equal the new red color value minus 20 (to a 
minimum of 0). By following the above steps, a stage one re-colored image is 
provided. Recall that the color values will range from 0 to 255. 

20 [0051] If the object is larger than an area threshold (such as, for 

example, an area threshold of 35 pixels), then the border of the object may be 
smoothed with the surroundings so that it appears more natural. These steps 
are illustrated in Figure 6. 

[0052] An original object 300 is shown in Figure 6a. This original object 
25 300 is dilated as described above to yield the dilated object 302 shown in 6b. 
This original object 300 is also eroded, as described above, to yield an eroded 
object 304 shown in Figure 6c. This eroded object 304 is then subtracted 
from the dilated object 302 of Figure 6b to yield a border region 306 shown in 
Figure 6d. 



I 
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[0053] The stage one re-colored image described above is then 
smoothed by convolving the image with a smoothing filter. For example, a 
possible smoothing filter is: 

12 1 

5 2 4 2 +16 

1 2 1 

Applying this filter produces an image designated the smoothed stage one re- 
colored image. The smooth stage one re-colored image pixels that are in the 
border region 306 are subsequently used to replace the corresponding pixels 
10 in the stage one re-colored image to produce the final re-colored image. 

Semi-Automated Operation 

[0054] Referring to Figure 5, there is illustrated in a flowchart a method 
for the semi-automated detection and removal of red-eye in digital 
photographic images in accordance with a further aspect of the invention. This 
15 function is typically defined to operate on a sub-image of a larger image. The 
size of the sub-image for this function should be larger than the maximum 
height and maximum width of the red portion of the eye. Referring to Figure A, 
there is illustrated an image of a child 110, in which a sub-image 112 is 
indicated as a grey box surrounding the child's eye 1 14. 

20 [0055] Referring back to Figure 5, in step 120. a sub-image 112 is 

provided. This may be provided by a user tracing a box around the portion of 
the image to be analyzed. In step 122, the appropriate number of erosions 
and dilations to use for the tophat operation during segmentation is 
calculated. The number of erosions and dilations is proportional to the size of 

25 the sub-image 1 12. As described above, the size of the sub-image 112 should 
be larger by some multiple than the maximum height and maximum width of 
the red-eye portion of the eye 114. This multiple can range from a lower limit 
to an upper limit. In step 122, the average of the width and the average of the 
height of the sub-image 1 12 are added. This sum is then divided by the upper 

30 limit and then divided by 2 and rounded to the nearest integer. This 
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represents the lowest number of erosion and dilation operations. The sum is 
then divided by the lower limit and then divided by 2 and rounded to the 
nearest integer. This represents the highest number of erosion and dilation 
operations. Then, a number of tophat operations are performed in step 124. 
5 The number of erosions and dilations used for each tophat operation ranges 
from the lowest number to the highest number. Similar to step 24 described 
above, small compact red objects are segmented in step 124 by performing at 
least one tophat operation as described above. This produces a segmentation 
mask where the locations and shapes of the segmented objects are 
10 represented by white pixels and all other pixels are black. 

[0056] In step 126, and analogous to step 26 described above, features 
of segmented objects are extracted for the objects in the segmentation block. 
For example, in step 128, analogous to step 28, each segmented object is 
classified based on the feature set extracted for that object in step 126. Again, 

15 as described above, the classification technique used is based on the object's 
quadratic classifier distance. If the quadratic classifier distance is greater than 
a given threshold, then the object is classified as red-eye. If the quadratic 
classifier distance is less than a given threshold, then the object is classified 
as a non-red-eye. By this means, a red-eye mask is produced in step 128. In 

20 step 130, analogous to step 52 described above, the areas classified as red- 
eye are re-colored using greyscale to produce the corrected image. This 
coloration may be the same as described above in connection with step 52. 
Alternatively, the greyscale values used may be equal to the average of the 
actual green and blue pixels in the original image. The features extracted 

25 describe each segmented object's original color, segmented shape and 
original color texture. 

[0057] Referring to Figure 7, there is illustrated in a block diagram, a 
computer system suitable for implementing an embodiment of the present 
invention. Specifically, the computer system comprises a CPU 210 connected 
30 to a RAM 212. The CPU 210 is also connected to an input/output controller 
208. which controls access to a keyboard 202. mouse 204 and monitor 206. 
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[0058] In accordance with an aspect of the invention, the CPU 210 is 

configured to implement a preferred embodiment of the invention. In this case, 
a digital photograph image would be stored in RAM 212, and, optionally, 
displayed on monitor 206. Then, the CPU 210 would perform steps on the 
5 digital photographic image, analogous to the steps described in relation to 
Figure 1 . to locate and correct red-eye. Alternatively, a user could use mouse 
204 to draw a box around a sub-image including instances of what is believed 
to be red-eye. Then, the CPU 210 could implement the method described in 
connection with Figure 5. 

10 Decrease Sensitivity Operation 

[0059] After automated correction of red-eye objects, the corrected 
image is shown on the monitor 206. At that point, a user may decide that 
some objects have been erroneously identified as red-eye and re-cplored. For 
example, a red Christmas tree light may accidentally have been re-colored. In 
15 general, a false positive object is a single object in one of the red-eye masks 
that does not correspond to an actual red-eye. 

[0060] Using a suitable input device, such as the keyboard 202 or 
mouse 204, the user can then trigger a decrease sensitivity operation. The 
red-eye mask images generated by the CPU 210 are retained in RAM 212 

20 until, at least, the next time an image is corrected. Also retained in RAM 212 
is the probability that each segmented object in the digital image displayed is 
red-eye. \Nhen the decrease sensitivity operation is triggered by the user, of 
all of the objects classified as red-eye from all of the segmentation masks, a 
specified number of objects with the lowest probability of being red-eye will be 

25 classified as non-red-eye, removed from the appropriate red-eye mask image 
and returned to their original color in the corrected image shown in the 
monitor 206. Optionally, a user may change the specified number of objects, 
such that the number of objects reclassified from being red-eye to being not 
red-eye changes. Alternatively, a user may simply change the threshold for 

30 the probability that an image is red-eye by raising the threshold, such that 
fewer objects will be classified as red-eye. 
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According to a further aspect of the invention, the user identifies a particular 
object identified as red-eye, which the user believes is not red-eye. The CPU 
210 will then raise the probability threshold required to classify an object as 
red-eye by an amount sufficient to declassify the object selected as red-eye, 
5 as well as any other objects having the same or lower probability of being red- 
eye. 

Increase Sensitivitv Operation 

[0061] When a corrected image is displayed in the monitor 206, a user 
may see an instance of red-eye that was not corrected. As described above in 

10 relation to the decrease sensitivity operation, segmentation masks images 
generated by the CPU 210 are retained in RAM 212 as is the probability that 
each object in the segmentation mask images is a red-eye. \A^en the 
increase sensitivity operation is called by a user, of all of the objects in all of 
the segmentation masks that have not already been classified as red-eye, a 

15 specified number of objects with the highest probability of being a red-eye will 
be classified as red-eye and added to the appropriate red-eye mask. The new 
corrected image will then include all of the red-eyes added by the increase 
sensitivity operation as well as all of those originally classified as red-eyes. All 
newly classified instances of red-eye will be re-colored as described above. 

20 Optionally, a user may select the number of objects to be re-classified. 
Alternatively, the user may directly lower the threshold probability required for 
an object to be classified as red-eye, or may pick out a particular instance of 
an object that, in the user's view, should have been classified as red-eye, but 
was not in the original corrected image, and then lower the threshold 

25 probability for an object to be identified as red-eye to a sufficient extent to 
enable the object selected to be identified as red-eye. 

Manual Override Operation 

[0062] A user viewing a corrected image in the monitor 206 may see a 
red-eye that was not detected, or an object that has been incorrectly classified 
30 as red-eye and re-colored. Further, it may not be possible to correct this 
problem using either the decrease sensitivity operation or the increase 
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sensitivity operation as doing so would create additional false negatives or 
false positives. In this case, the user may indicate the pixels of either the 
undetected red-eye or the false positive object using the mouse 204, say, and 
then call the manual override operation. 

5 [0063] When the manual override operation is called, it determines if 
the coordinates identified by the user are located on an object currently 
classified as red-eye. If so, then the user considers this to be a false positive, 
and the operation will remove the object from the appropriate red-eye masks 
and set the probability that the segmented object is a red-eye to a lower value 
10 than the probability for any other segmented object in the entire image. This 
indicates for the purposes of any further operations that this object is unlikely 
to be a red-eye. Finally, the object is returned to its original color in the 
corrected image. 

[0064] If the coordinates indicated by the user are located on an object 

15 currently classified as not red-eye, indicating that the user considers this 
object to be an undetected red-eye, then each segmentation mask is checked 
to determine if the coordinates are located on a segmented object that is 
present in the mask. If so, then that object will be classified as red-eye and 
added to the appropriate red-eye mask. The probability that the segmented 
20 object is a red-eye is then set to a higher value than the probability for any 
other segmented object in the entire image. This indicates for the purposes of 
any further operations that the object is highly likely to be red-eye. Finally, the 
object is re-colored in the corrected image as described above. 

[0065] Other variations and modifications of the invention are possible. 

25 For example, while the foregoing has been described in the context of a red- 
green-blue pixel coloring system, other color systems could be used, such as, 
for example, a cyan-magenta-yellow-key system or a hue-saturation-value 
system, which similarly represent colors as combinations of their respective 
color components. All such modifications or variations are believed to be 

30 within the sphere and scope of the invention as defined by the claims 
appended hereto. 



